Frequency in Morphology

0 Introduction The recent w ork in statistical parsing Church 1988, Schabes 1991 and statistical machine translation Brown et al 1990 calls the traditional rule-based view of grammar i n to question. These authors emphasize that grammatical rule systems aiming at syntax-directed translation, and even rule systems aimed at the description of a single language, break down when faced with the actual complexity of natural language data. In fact, under realistic testing conditions the example-based" or corpus-based" systems that employ some general-purpose optimization algorithm in order to extract statistical regularities from the data fare just as well as the rule-based systems in which the regularities are extracted beforehand by the grammarian. In the light of these facts it is natural to extend the inquiry to morphology and ask how statistical morphological systems that exploit the frequency information in the data will compare with rule-based morphological systems that exploit the expertise of the grammarian. The paper reports the results of a pilot study performed on the largest extant machine-readable corpus of a morphologically complex language, namely the SZO1TA1R corpus see Kornai 1986 based on the Debrecen Thesaurus Papp 1969 and the Frequency Dictionary of Hungarian F uredi and Kele-men 1989. Section 1 presents the necessary background information about statistical approaches to grammar in general, about the traditional position-class view of morphology and its modern generalizations Kiparsky 1982, Koskenniemi 1983, and about the Hungarian nominal and verbal paradigms Antal 1961, 1966. Section 2 presents a simple statistical method to estimate the number of innected word tokens required for saturated paradigms" i.e. stems for which all paradigmatic forms are actually attested in the corpus. Although the SZO1TA1R corpus is to small to contain fully saturated paradigms, using an empirical nding of the pilot study, namely that linguistically orthogonal mor-phosyntactic features are statistically independent, we can still estimate the required corpus size. In order to saturate the paradigms of the most frequent 1,000 verb stems we need to collect a corpus 630 to 800 times bigger than the present one which is .5m words, and in order to saturate the paradigms of the 30 most frequent nouns we w ould need over 33 billion words. Do we really need all the paradigmatic forms to be exempliied in our corpus to be able to build them into our statistical morphology? The concluding Section 3 proposes hybrid Hidden Markov Model HMM systems that bring both grammatical …